Quality assessment in undergraduate medical training: how to bridge the gap between what we do and what we should do

Introduction the outcome of the undergraduate medical training programme in South Africa is to produce competent medical doctors who can integrate knowledge, skills and attitudes relevant to the South African context. Training facilities have a responsibility to ensure that they perform this assessment of competence effectively and defend the results of high-stakes assessments. This study aimed to obtain qualitative data to suggest practical recommendations on best assessment practices to address the gaps between theoretical principles that inform assessment and current assessment practices. Methods a focus group interview was used to gather this data. The teaching and learning coordinators for five of the six modules that are offered in the clinical phase of the undergraduate medical programme participated in the focus group interview. The focus group interview proceeded as planned and took 95 minutes to complete. The responses were transcribed and recorded on a matrix. Results the lack of formal feedback to students was identified as an area of concern; feedback plays an important role to promote student learning and improve patient care. The role of teaching and learning coordinators as drivers of quality assessment were recognized and supported. All participants agreed on the outcome of the programme and the central role of the outcome in all assessments. Conclusion the training of assessors and the implementation of workplace-based assessment and assessment portfolios were recommended and can also address feasibility challenges. Participants recommended decreasing summative assessments and only performing these for borderline students.


Introduction
Quality assessment requires that the type and content of the assessment is aligned with the outcome of the training programme [1]. The outcome of the undergraduate medical training programme in South Africa is to produce competent medical doctors who can integrate knowledge, skills and attitudes relevant to the South African context [2]. Assessment of clinical competence is a complex process, due to a number of factors, which include the constant emergence of new best-practice medical evidence [3], the theory-practice gap between what is taught and what is observed in clinical practice [4][5][6], what is feasible [7], and the challenges of assessment in real-life situations that may compromise the reliability of the assessment [8]. Competence assessment must satisfy various stakeholders, which include patients and the general public, training providers, regulatory bodies and students.
Training facilities have a responsibility to ensure that they perform this assessment task effectively and can defend the results of high-stakes assessments [9]. A paper describing a framework to benchmark the quality of clinical assessment in a South African undergraduate medical programme, provides context-specific theoretical principles for undergraduate medical assessment [10]. Assessment reports and quantitative studies (In press) on current assessment practices used for undergraduate medical students at the University of the Free State (UFS) showed that these principles are not always adhered to, which may compromise the defensibility of high-stakes assessments. This study aimed to obtain qualitative data to suggest practical recommendations on best assessment practices to address the gaps between theoretical principles, that inform assessment, and current assessment practices. These recommendations will be combined with other research results to prepare a proposal to inform quality assessment at the UFS.

Methods
Research design: a focus group interview (FGI) was used to triangulate theory (i.e. theoretical principles that inform assessment) with current assessment practices, to compile recommendations that should assist with quality assessment in undergraduate medical training. An FGI can be used in a mixed-methods design to triangulate qualitative and quantitative data from different sources [11], as was done in this study. Various definitions exist for an FGI, and some researchers even use the terms FGI and focus group discussion (FGD) interchangeably [12]. The difference between an FGI and an FGD is that the main objective of an FGI is to obtain answers to specific questions while, in an FGD, the interaction between the group members and the group dynamics are as important as the information gathered [12,13].
Merton and Kendall (in Cohen et al.) [14] first described the concept of an FGI in 1946 and concluded that: during an FGI, there is a greater degree of interviewer control; the people participating in the interview should share experiences; the interview questions are based on previous data analysis; and subjective experiences of people who have been exposed to the same experience are gathered. The strength of a focus group is that it stimulates new or forgotten ideas and that members can build on the input of others. Some of its limitations are that it can be difficult to get members together, the group may not be representative, and some group members may dominate others [14,15].
Participants: in an FGI, between five and 12 members interact, debate and argue their opinions on a specific issue. The participants of the focus group should represent the target population. Members that participate should do so voluntarily, should be knowledgeable on the subject and able to communicate in a group [11]. The clinical phase at the UFS comprises six modules. The six teaching and learning (T&L) coordinators of these modules were invited to participate in the FGI. Five of these T&L coordinators participated in the FGI.
Facilitator: the facilitator asks specific questions with the view to obtain answers to specific questions [13]. It is important for the facilitator to monitor the group dynamics and ensure participation by all members. The facilitator must be in control of the situation and should avoid too much or too little personal participation [12]. A facilitator with experience in higher education and in conducting FGIs was used to facilitate the process.
Questions: an FGI is not merely a general discussion, but is focused on a specific topic. Usually, the discussion starts broadly and, then, spirals inwards to address the research question/s [16]. The questions asked during this FGI derived from an assessment framework for undergraduate medical programmes [10], as well as the results of current assessment practices (In press) and publications with recommendations for undergraduate medical assessment [1,2,9,17]. The guidelines for developing "good focus group questions", which include that the questions must be short, clear, open-ended and directional, as described by Krueger and Casey [18] were followed. Questions were categorised and grouped.
All the questions were available in the facilitator and participant guides which the facilitator and participants received before the FGI.
Logistics: an FGI should last between 60 and 90 minutes [19]. To capture all the information, the facilitator needs to take notes of the discussions and non-verbal cues. It can be helpful to record or videotape the discussion, and to use a co-facilitator to take notes and write down observations too [12]. The researcher arranged a neutral venue, confirmed the availability of the facilitator and participants and provided refreshments. The facilitator received all the necessary documents well in advance of the FGI. The researcher met with the facilitator in person about the process to be followed and to clarify uncertainties and agreed on the process. All participants received a participant guide one week before the FGI and a reminder to attend one day before the FGI was conducted on the 29 th January 2020.
Data collection: the aim of an FGI is not consensus, but rather the gathering of rich ideas [11]. The facilitator asked one question at a time and encouraged active participation by all participants.
Discussions continued until all participants were satisfied with the answer to a particular question. If no answer or more than one answer or suggestion were offered, the facilitator encouraged participation until no new ideas were produced. More than one answer or disagreement between opinions were allowed.
Pretesting of focus group and explorative interview: no test run of the FGI was done, as it is important to obtain the collaborative feedback of the whole group. The validity of the questions asked in the FGI was discussed in an explorative interview with the promotors, and was based on previous experience of the researcher.
Analysis of data and reporting: an audio recording of the FGI was transcribed by the researcher immediately after the FGI concluded.  Participants were not identified and a participant number was allocated to each, which is also used for data reporting.
Quality and rigour of the data management: to ensure the credibility of the data collection, all the research questions were clarified with the promotors. The facilitator ensured active participation by all participants, and clarified concepts to improve the quality of the data. Local, national and international assessment guidelines were included to make the recommendations transferable to other institutions. The focus group participants and interview process were clearly described for the purpose of assessing the dependability of the results. Confirmability was ensured by audio and video recording of the process and verifying results after completion of the result template.

Results
The T&L coordinators for five of the six modules that are offered in  Table 1 the results for the outcome of the programme, competence, validity and reliability are displayed. Table 2 addresses the results for fairness, feasibility, educational effect and assessment methods and Table 3 quality assurance, training and general comments.

Discussion
The FGI met the requirements for a good FGI regarding participants, the facilitator, the questions, logistics, explorative interview and data collection and analysis. The results are also representative of the study population, with five of the possible six participants included. The first question was around the outcome of die undergraduate medical programme. All the participants agreed with the outcome as is, namely, to produce a competent medical doctor who can integrate Clinical competence must be assessed on the "Does" level, according to Miller's pyramid [22]. It was mentioned that the actual demonstration of this competence only occurs during internship, which is still part of training (students must complete internship and community service before registering as independent medical practitioners with the Health Professions Council of South Africa (HPCSA). A suggestion to implement pass/fail stations and not only an average of 50% or above to pass, was well accepted. A discussion on the difficulty to ensure competence with a pass mark of 50% (the pass mark according to the UFS assessment policy) provided more questions than answers. It must be recognised that a mark of 50% indicate that the student is competent and not "half competent". All assessors should be aware of how they allocate marks and the implication thereof. Further discussion in this regard was recommended to clarify the meaning of 50% in the context of competence.
During questions regarding validity, good practices were shared and recommendations made. It was agreed that T&L coordinators should take responsibility for assessments, to ensure the validity of assessments. Blueprinting of all assessments should be done.
Blueprinting will improve content validity, and using appropriate assessment methods will improve construct validity [10]. There is no need to add additional assessment methods, as most assessment methods described for undergraduate clinical assessment [23] are currently used at the UFS. It was recognized that a shortage in the workforce favours the use of less labour-intensive assessment methods, e.g. multiple-choice questions rather than longer written questions that can assess higher cognitive levels. The lack of trained assessors also limits the use of workplace-based assessment (WBA) and assessment/competency portfolios to assess competence. To address the workforce issue, all clinicians should be trained as assessors, and registrars can be included in the assessment process.
By including registrars, they are trained on the important skill of assessment, and it may help to spread the workload. Regarding the assessment of professionalism and "soft skills," the suggestion to implement a "professionalism portfolio" and implement the graduate attributes policy of the university were supported and should be investigated.
The participants gave valuable input on aspects to improve the quality of assessment, including recommendations on reliability, fairness, educational effect and feasibility. Competency assessment cannot be 100% reliable, but the suggestions to use WBA and assessment/competency portfolios were recommended to increase the number of assessments. WBA and assessment portfolios are excellent ways to assess competence, but reliability may be compromised [24]. Although portfolios and WBA are labour intensive, these methods are more authentic and the number and type of assessments can increase, thereby contributing to reliability [25]. The lack of formal feedback to students was identified as an area of concern -feedback plays an important role to promote student During the FGI, clinical training was also discussed in relation to assessment. Biggs [29] describes the term constructive alignment as comprising outcomes, teaching and training activities and assessment that are planned to complement and support each other. Students indicated in their feedback before the FGI that they want more on-site practical training in wards and clinics (In press). The increase in student numbers and decrease in teacher numbers also decreases supervised, hands-on practical training for students. A suggestion for countering the lack of clinical exposure is to stipulate clearly and monitor available clinical training time. Another factor that affects clinical training negatively is overburdened clinicians, who may not necessarily be good role models and tend to give students time off, so that the clinicians can get clinical work done, rather than spend time on training. This practice may be due to burnout, as evidenced by a study in this academic setting that showed that only 3. The training platform may be an opportunity for students to see how to behave professionally, but also how not to behave. It was discussed that students may not be aware that, although they are trained in tertiary facilities, they are not expected to perform as specialists, but that they should rather use the opportunity to identify clinical signs and develop an approach to a specific problem. Better communication on the outcome of specific training rotations may assist both students and clinicians and was recommended. The FGI concluded with a discussion on the effect of the introduction of T&L coordinators on student assessment and training. The excellent work of the T&L coordinators was recognised and appreciated. All agreed that the T&L coordinators should continue to play a leading role in student assessment and training.
Limitations and strengths: only the T&L coordinators of the major disciplines participated in the FGI, and the FGI may have failed to capture contributions by excluding minor disciplines. However, these smaller disciplines were indirectly represented by the major disciplines. Strengths of the FGI were that the FGI was conducted according to the planning, and within the guidelines for a FGI, as described in the methods, and that data management met the criteria for credibility.

Conclusion
The clear, agreed-upon outcome, namely, to produce a competent medical doctor who can integrate knowledge, skills and attitudes relevant to the South African context, should be kept in mind during all assessments. The difficulty of how to measure and allocate marks to competence was recognised. The lack of formal feedback to students and blueprinting should be addressed. The important place of WBA and assessment portfolios, with less emphasis on summative assessment were important recommendations from the FGI. A proposal to improve the quality of assessment in the clinical phase of the undergraduate medical programme will be compiled from this and other research information. This proposal will be submitted to the Executive Committee of the School of Clinical Medicine for implementation. Finally, an FGI can be recommended as an appropriate way to get rich data for practical solutions.

What is known about this topic
• Assessment should be aligned with the outcome of the training programme; • Assessment of clinical competence is a complex process.

What this study adds
• Workplace-based assessment should form part of competency assessment; • The difficulty of how to measure and allocate marks to competence was recognised; • Competency and professional portfolios should be implemented.

Competing interests
The authors declare no competing interests.

Authors' contributions
HB: conceptualisation of study, protocol development, data collection and writing of paper, JB and LJVdM: promotors who assisted with conceptualisation and planning of the study, as well as critical evaluation and final approval of the manuscript.

Acknowledgments
Prof Mathys Labuschagne for facilitating the focus group interview.
Mrs Hettie Human for language editing.    Discussion: It is difficult to achieve competence before you start to work, and Internship is also part of training and becoming competent. Students are generally competent, but I don't think that we assess competence well enough. 50% is not necessarily a mark that indicates competence. We should ensure that if a student gets 50% that the student is competent and not "half competent", as 50% is the pass mark. This should be discussed at other forums Discussion: Assessors need to be trained better, e.g. registrars can be trained in assessment, by allowing them to assess together with a consultant and then discuss the marks. It will benefit both parties and address the work force.

According to the Health Professions Council of South
Africa (HPCSA) "soft skills and professionalism" should be assessed. How do you suggest that we assess "soft skills" and professionalism throughout the curriculum?
3.3.1 Soft skills are assessed in clinical case presentations, but a specific mark is not allocated to it. We may allocate a specific mark to it  Discussion: The outcomes should be a framework, rather than specific. This is tricky, because we need specifics to blueprint. The students struggle with the transition between pre-clinical with specific outcomes and clinical training with broader outcomes. They need to mature in this regard. These are senior students and we should not spoon feed them. They must be able to integrate and think rather than concentrating on detail 5 Discussion: This is the best place to assess real-life competence. The students, patients and assessors are there. Peer assessment may be problematic. The use of a "competency portfolio" was suggested and supported. Discussion: This will increase the number and the reliability of assessments. You know which students are competent when you work with them. You can also assess professionalism better. The competency portfolio was mentioned again.

What is your opinion on summative assessment?
Although more assessments are good learning opportunities, I think we must try and reduce summative assessment to only borderline candidates.

A OS A A A
Discussion: Students may be disadvantaged during their first rotation, because they gain experience and competence throughout the year. However all will have first rotations and it is therefore fair. Discussion: Most students want to go home as soon as possible, stating that they want to study, which is contradicting what they suggested.
10.2 Students also suggested more exposure to good clinical role models. What is your response to this?
10.2.1 All clinicians are not necessarily good role models, but students can also learn from the "not so good" on what not to do.

OS A A A A
Discussion: Due to workload many people are suffering from burnout. This must also be discussed with students and the importance of self-care must be re-emphasized. The clinical psychologist can assist and attention should be paid to resilience training. Reflective practice and professionalism must also be addressed. Students get very good support at the UFS to cope with stress.